discovery-index(query): stop caching a GitHub-failure-truncated candidate set for the full TTL and serving it as complete - #10243
Conversation
…date set for the full TTL and serving it as complete Fixes JSONbored#10031
|
Tip ✅ LoopOver review result - approve/merge recommendedReview updated: 2026-07-31 14:06:24 UTC
Review summary Nits — 5 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10243 +/- ##
==========================================
+ Coverage 80.57% 80.60% +0.03%
==========================================
Files 283 284 +1
Lines 59095 59195 +100
Branches 6996 7026 +30
==========================================
+ Hits 47614 47714 +100
Misses 11188 11188
Partials 293 293
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Summary
fetchRepoIssuesandsearchIssuesabandon pagination on the first non-OK page and return the pages theyalready collected, with a warning (
packages/discovery-index/src/github-client.ts:189-203):searchIssues(packages/discovery-index/src/github-client.ts:214-231) has the identical shape. The callerlogs the warning and otherwise ignores it (
packages/discovery-index/src/discovery-query.ts:176-178):Two things then compound it:
runDiscoveryQuerycaches whatevercomputeCandidatesreturned, for the full TTL(
packages/discovery-index/src/discovery-query.ts:213-216):A page-2 500 during one cold miss therefore pins a truncated candidate set for
DEFAULT_CACHE_TTL_MS(300 s,
packages/discovery-index/src/discovery-query.ts:40) for every caller sharing that scope. Thewhole point of the service is that many miners share one cached scope, so one transient GitHub failure is
amplified across the fleet rather than isolated to the request that hit it.
The response then asserts completeness.
runDiscoveryQueryemitsnextCursor: nextOffset < allCandidates.length ? encodeCursor(nextOffset) : null(
packages/discovery-index/src/discovery-query.ts:218-221), so a client walking the cursor reachesnextCursor: nulland correctly concludes it has seen the whole set — which it has not. Nothing on thewire distinguishes "this scope really has 40 candidates" from "GitHub 500'd on page 2 and we have 40 of
200". The only signal is a
console.errorwarn line inside the server process.This is the same class of gap the main app already closed for its own truncated GitHub reads:
fetchRepoTree'sconsumer treats a truncated tree as inconclusive and returns
nullrather than a short list(
src/github/migration-tree.ts:41-44), andfetchPullRequestFilespushes an explicit truncation warning intothe review's own warnings so the PR is held rather than silently evaluated on a partial file set
(
src/github/backfill.ts:2469-2476).Deliverables
computeCandidatesreturns both the candidate list and a completeness flag (or equivalent) derived fromthe warnings already collected at
packages/discovery-index/src/discovery-query.ts:176-178,:183-185,:190-192.runDiscoveryQueryskips theresultCachewrite when the pass was incomplete. Exact expectation: with aGitHubClientLikestub whosefetchRepoIssuesreturns{ issues: [oneIssue], warnings: ["GitHub returned 500 for owner/repo issues"] },two successive
runDiscoveryQuerycalls for the same query BOTH invokefetchRepoIssues(i.e. thesecond call is a cache miss), and both responses carry the one candidate.
test/unit/discovery-index/discovery-query.test.tsasserting the unchanged happy path: a stubreturning
{ issues: [...], warnings: [] }is called exactly once across two successiverunDiscoveryQuerycalls for the same query, and the second response is identical to the first.test/unit/discovery-index/discovery-query.test.tscovering thesearchIssueswarning path(an
orgs-scoped query) as well as thefetchRepoIssuespath — both feed the same completeness flag anda partial fix could cover only one.
test/unit/discovery-index/discovery-query.test.tsnamed for this bug (e.g."REGRESSION: a GitHub-failure-truncated candidate set is not cached for the TTL").All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example one
that threads the flag through
computeCandidatesbut still callsresultCache.getOrCompute, so the truncatedset is written before the flag is ever consulted — does not resolve this issue.
Test plan
This repo enforces 99%+ Codecov patch coverage, branch-counted.
vitest.config.ts'scoverage.includecovers
src/**/*.tsandpackages/loopover-engine/src/**/*.ts—packages/discovery-index/src/discovery-query.tsis not in
coverage.include, socodecov/patchdoes not gate this path. The tests above are stillmandatory:
test/unit/discovery-index/discovery-query.test.tsalready exists and runs innpm run test:ci,and the deliverables are only verifiable through it.
Both arms of every new branch must be tested: warnings-present vs warnings-empty for the repo path, the same
for the org-search path, the same for the search-terms path, and the cache-write vs cache-skip decision in
runDiscoveryQuery. ThegetOrComputeshort-circuit (cached value present) and the compute path must bothstill be exercised.
This change is NOT in
packages/loopover-engine/src/**, so the dual-upload engine-coverage rule does not apply.Fixes #10031